Overview

This article provides an overview of the Lotus Expeditor integrator platform, the architecture, the various components and how they stack up, and how the Integrator plugs into other hardware and software components. Figure 1 shows an architectural overview diagram example including the nearby systems of an overall IBM Lotus Expeditor integrator solution for retail enterprises. The solution example is based on existing retail customer requirements and business use cases which drove the development of Expeditor integrator based on the IBM® Lotus® Expeditor Client for Desktops product.

Figure 1 - Lotus Expeditor integrator Architectural Overview example for retail solutions

The Lotus Expeditor integrator solution can extend the reach of an existing Application Integration back-end based on a common Backbone Service Bus (e.g. WebSphere® Message Broker and WBI Adapter for Flat Files) to remote locations with restricted hardware and software resources.

In the given retail example, data exchange between the 4690 point of sales controllers and the Backbone Service Bus is usually done via FTP and file system access today. Expeditor integrator replaces the existing process by providing:

• Reliable data exchange between enterprise Backbone Service Bus and remote locations (e.g. stores).
• Centralized management (monitoring, configuration management & software distribution).
• Near zero operating effort for the integration infrastructure at the remote location (e.g. store infrastructure such as PoS and scale systems).
• Local file access support for local and mounted file systems as well as FTP, FTPs, SSH protocol support at the remote location.
• Local data access through JDBC and REST data adapters at the remote location.

The Expeditor integrator was developed based on open standards (for example, OSGi®, Eclipse®).

Figure 1 also shows the core components of IBM Lotus Expeditor integrator. The main processing unit is the Expeditor integrator Access Control Service (ACS) which manages Flows of Activities. These flows can be manually configured to fulfill distinct (business) use cases. The Resource Monitor monitors given resources of different types through the different Resource Adapters (e.g. FTP Adapter). It fires events for triggering ACS Flows for configurable states of a given resource (e.g. when a defined file appears in a defined directory an event is published which triggers an ACS Flow). The IBM Lotus Expeditor micro broker is the message provider of the Expeditor integrator which ensures performing, reliable and secure data exchange with the back-end or other local resources. Monitoring Services create (configurable) standards based CBE events which can be either sent to the back-end messaging system or to other (business) monitoring systems such as IBM Tivoli Enterprise Console and IBM Business Monitor.

Component Overview

The modular design of the Expeditor integrator is based on the OSGi® and the Eclipse® Standard (www.osgi.org, www.eclipse.org). This approach was chosen in order to achieve the required stability of a remotely running application, to address limited available resources on the remote server (e.g. a retail store server) and to get remote control options.

Figure 2 shows the components of the Expeditor integrator.

ACS (Expeditor integrator Application Control Service)
The ACS is the core component of the Expeditor integrator. The Expeditor integrator Application Control Service is a (more general) central processing component for all incoming and outgoing requests/messages in Expeditor integrator. It interprets the requests/messages and triggers further actions to orchestrate a use-case within a transactional unit of work. The successful execution of all ACS Activities within the use-case flow leads to the committing of the containing unit of work within the Expeditor integrator’s Transaction Service or a rollback otherwise.

The ACS Service has been designed to be both flexible and simple to operate and with possibilities for future extension. To achieve this, the ACS uses a simple flow language provided in XML files which indicate a series of Activities to be executed in a sequence. This allows the flows to be configured without necessarily having to write code.

The ACS implements the functional requirements for Expeditor integrator based (custom) solutions. It provides data transmission capabilities for the integration of systems which are connected to the Expeditor integrator runtime machine (e.g. PoS systems connected to the retail store server). In the retail scenario this would include for example:

• Transmitting files as messages reliably from the Store Server/4690 POS Controller to the AI messaging back-end (e.g. transaction log files)
• Transmitting files as messages from the AI back-end to the Store Server and/or the 4690 POS Controllers (e.g. price update files)

These functions must be put in a reliable transactional context.

Figure 2: Expeditor integrator components

IBM J9 DRE
The IBM J9 is a Virtual Machine that implements the Java specification (Development Runtime Environment available through IBM Lotus Expeditor for Desktops). It is available for a variety of operating systems.

The Lotus Expeditor Client for Desktops is shipped with jclDesktop for Windows XP/Vista that is an optimized runtime environment for desktop use that enables Java developers to write applications in a "fit for purpose" runtime. There is also an optional runtime for JSE compliant program code that can be installed and licensed as a separate product. This optional Lotus Expeditor Client runtime DRE is used as base for the Expeditor integrator, because the IBM WebSphere® MQ® Client Java library and the JET require libraries that comply with the JSE standard.

OSGi® Service Platform
OSGi is a production-ready Java environment for running and managing systems and application bundles across applications in a single JVM. The OSGi model is instantiated in IBM Lotus Expeditor Client

Lotus Expeditor micro broker
The IBM Lotus Expeditor micro broker acts as a message hub for clients connected to it. These clients can send messages to each other. Clients can run in the same process (Virtual Machine (VM) that implements the Java™ specifications) as the micro broker, in a different process on the same computer, or even a different computer providing a powerful messaging infrastructure.

Custom Event Service (CES)
The CES is an independent component (bundle) that listens for OSGi Framework events that are fired by the Expeditor integrator bundles using OSGi Event Admin. The Custom Event Service monitors the OSGi Event Admin for certain configurable messages, converts them to Common Base Events (CBE) and forwards them to the AI messaging back-end of the Central Systems by using the Lotus Expeditor micro broker (for example, by using the Expeditor integrator’s EventQ, Business Events carrying standardized Common Base Events can be sent to the back-end).

Custom Log Service
All Expeditor integrator bundles are using the standard OSGi Log Service for logging status information. This enables them to run on every OSGi standard platform without needing an application dependent log service. The Custom Log Service is an independent component that recognizes log entries written by the Expeditor integrator bundles by listening to Log Events fired by the OSGi Log Service. Depending on the Custom Log Service’s configuration, certain log entries are forwarded by the Custom Log Service as Log Messages to the central back-end (Monitoring System), for example by using Expeditor integrator’s LogQ. The log entry is converted in the standard CBE format so that a monitoring process at the central back-end is able to retrieve these Log Events and provide detailed status and health information about the Expeditor integrator to the Administrator or other monitoring services.

The Expeditor integrator Tivoli Monitor Service (iTMS)
The iTMS component is operating in the same way as the Custom Log Service, but does not convert the captured Log entries into CBE and forward them to the configured messaging server in the back-end. The iTSM uses the Event Integration Facility (EIF), which is provided with the Tivoli® Enterprise Console (TEC). The EIF is exploited for the creation of TEC events out of the OSGi log entries and for forwarding them to the TEC.

Custom Config Service (CCS)
The CCS extends the OSGi Configuration Admin Service interface with the capability of retrieving configuration data through either

• the local Expeditor integrator configuration files (XPDinteg.xml) or through
• server-managed configuration updates using the OSGi Configuration Admin service directly (for example, IBM Lotus Expeditor server)

The use of the Expeditor integrator configuration files provides simple, but rather inflexible remote configuration capabilities. The advantage of storing configuration data through the OSGi Configuration Admin interface is that configurations can be centrally managed by the Lotus Expeditor Device Management Server or similar products during runtime (for example, IBM WebSphere Premises Server).

The Custom Config Service combines the advantages of both approaches by applying configuration changes in the XPDinteg.xml file to the OSGi Config Service. The Custom Config Service also logs status and execution information using the Custom Log Service.

The Resource Adapter Interface provides a common interface to the File System Adapter, the FTP/SSH Adapter, the JMS Destination Adapter, the Database/JDBC Adapter and other future adapters to deliver transparent resource access services. The Resource Rdapters can be registered with the Resource Monitor Service. The Resource Monitor Service uses the OSGi Event Admin service to inform interested parties (bundles) about certain events that occurred at the Resource Adapters (e.g. when a new message arrives or a file was created on the local file system).

(Local) File System Adapter
The File System Adapter provides simple file handling capabilities independently of the used file system. The connector provides the following functions:

• Retrieve a directory listing
• Write a file to the local file system
• Read file from local file system (regular expressions can be used to select files)
• Monitor files and directories

Enhanced FTP Adapter
The Enhanced FTP Adapter provides FTP client capabilities for transmitting files between Expeditor integrator runtime server using the FTP, sFTP or SSH protocol (e.g. between the retail store server and the 4690 POS Controllers). The FTP/SSH Adapter provides the following functions:

• Retrieve a directory listing of the remote directory
• Send a file to the remote file system (FTP/SSH put)
• Retrieve file from local file system (FTP/SSH get: regular expression can be used to select files)
• Monitor files and directories

Database (JDBC) Adapter
The Database Adapter provides access to local and remote databases which can be accessed with JDBC drivers. The Database Adapter offers:

• Read and write database records using SQL and Expeditor integrator command language (insert, update, delete records)
• Take XML file content and write into database with XML support; retrieve XML content form database and send as message to back-end
• Monitor database tables and records

Resource Monitor Service (RMS)
The RMS is responsible for monitoring resources local to the Expeditor integrator runtime and emitting particular events for consumption by the Expeditor integrator Application Control Service (ACS) and other interested components via the OSGi Event Admin service. Resources might represent files or directories on a file system, queues in a messaging server, database entries and so on. Operations over resources such as a read or a write are considered transactional and as such each operation provides an XAResource implementation so that the operation can be enlisted as part of a transactional unit of work. Within the Expeditor integrator system, there is a single Resource Monitor Service that monitors a number of Resource Adapters.

REST Adapter
The REST Adapter provides the optional HTTP access channel besides the messaging protocol to the Expeditor integrator runtime. Expeditor integrator offers a specific HTTP GET request schema which must be used for accessing the REST Adapter and invoking ACS flows.
The REST Adapter converts the incoming request data into Expeditor integrator messages and sticks them into any local integrator queue. From there on, the Expeditor integrator works in the same way as for incoming messages (and processes these messages).

Queue Dispatcher (QDP)
The QDP monitors the default Expeditor integrator incoming request queue (ReqInQ) for messages and dispatches them to local queues regarding to the type of message received. This avoids blocking of new threads by other running threads which are processing long message sequences (e.g. building a file from a sequence of messages which are received over a longer period of time). It also moves messages for the back-end system from the local queues to the bridged corresponding remote back-end queues.

The dispatching algorithm (for example, local target queues) is also configurable.

Optional: Expeditor Client Management:
The IBM Lotus Expeditor Client Management server allows for remote management of both Expeditor integrator features and plug-ins as well as resources such as configuration files. Additional tasks supported by the Expeditor Client Management server are documented in the Lotus Expeditor integrator Administrator's guide.

Example Business Use Cases for Retail Overview

For the initial Expeditor integrator release, three main Business Use Cases were especially addressed:

• BUC_1: Master Data Load/Update
  Master data is sent by the back-bone service bus (BSB at the back-end) to the store (e.g. price updates for PoS systems).
  
  There are business flows in the BSB which create messages carrying data for the PoS systems of the stores in their payload. These messages are once-and-only-once delivered to the Expeditor integrator running at the remote location (e.g. on the retail store server). Then, the payload data of the messages need to be extracted from the messages, written to files in a given format, encoding etc. and must be finally transferred to their endpoint. In this use case, the 4690 FTP Server is the final destination for the transferred files. Other endpoints could be the local file system (see BUC_3), any SSH File Server or a database.
  
• BUC_2: Transaction Data Transfer
  The back-end service bus (BSB at the back-end) receives files from the remote location stores (e.g. files containing sales transactions are retrieved from a retail store).
  
  This use case represents the previous use case in inverse order. The existence of a given file(s) or database record triggers their retrieval through the Expeditor integrator. The Expeditor integrator takes the file (database record) content and places it into messages for the back-bone service bus. These messages are then securely transferred from the remote location (e.g. retail store server) to the back-end queue where they can trigger further business processing.BUC_3: Scale/Label Data Transfer
  
  Data for scales and label printers is sent from the back-end service bus to the remote location (retail stores).
  
  This use case is similar to the first one. The major difference is that the final endpoint for the files, which are created by the Expeditor integrator out of the received BSB messages, is the Store Server’s local file system.

These use cases are created by defining combinations of Expeditor integrator Application Control Service Activities (ACS Activities). These combinations are referred to as “Flows” which are executed in the Expeditor integrator Application Control Service. The flow configurations are stored in flow definition files and can easily be adapted to new business scenarios. Each flow is triggered by a given event that can be fired by another application component (for example, a certain Resource Adapter.

Furthermore, the Expeditor integrator offers a set of ACS Activities that can be linked within flows to create new use case implementations (see Building the Flow Definitions for Business Use Cases for more details).